On Computing Nash Equilibrium in Stochastic Games

Using the fact that any two player discounted stochastic game with finite state and action spaces can be recast as a non-convex constrained optimization problem, where each global minima corresponds to a stationary Nash equilibrium, we present a sequential quadratic programming based algorithm that converges to a KKT point. This KKT point is an -Nash equilibrium for some > 0 and under some suitable conditions we show that this KKT point corresponds to a stationary Nash equilibrium. The algorithm updates the Hessian matrix of the Lagrangian function in a specific way. We illustrate various difficulties that can arise while computing stationary Nash equilibrium of the stochastic game using a variant of pollution tax model. One interesting feature of this model (in an instance) is that it admits a Nash equilibrium which is independent of the discount factor close to 1, an extension of Blackwell optimality in Markov decision processes.